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Abstract 


This paper explores different approaches to the sonification 
and visualisation of two environmental projects:- “Heavy 
Metal” is focussed upon the real-time analysis and sonifica- 
tion of the chemical elements in a painting via a camera vision 
system, whilst “Oratorio for a Million Souls” concerns the be- 
haviour and acoustic properties of live bee colonies manifest 
in the creation of real-time multi-channel sound compositions 
and associated sound architectures. Whilst these two projects 
differ in terms of methodology, aesthetics and technical ap- 
proach they both share a direct concern with a deep analysis 
of the underlying environmental structures and perceptual 
frameworks that emerge in direct ‘live’ encounters — on one 
hand the discovery of what lies behind the surface of a painted 
image in terms of chemical and colour structures — and in 
the case of Oratorio, a compelling immersion into the acoustic 
environment of Bees. 
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Introduction and context for Heavy Metal 


Heavy Metal' (2016) is an art installation with interactive 
camera vision system and digital audio by artist Nigel 
Helyer in collaboration with composer Jon Drummond and 
Environmental Scientist Mark Taylor. Heavy Metal was 
conceived and realised as part of a three-year Australian Re- 
search Council Linkage Grant project, When Science Meets 
Art: an environmental portrait of the Shoalhaven River Val- 
ley. A creative research collaboration between artist Nigel 
Helyer, environmental scientist Mark Taylor, and media 
theorist John Potts. 


The aim of the overall project is to create a complete envi- 
ronmental portrait of Bundanon’, a region of 1100 hectares 
(2700 acres) in rural NSW bequeathed to the Australian peo- 
ple by the modernist painter Arthur Boyd? in 1993. The pro- 
ject uses techniques of environmental science, artistic prac- 
tice, information technology, media technology and cultural 
history. Data representing environmental quality at Bun- 
danon is digitally transformed into visual information and 
sound, and communicated by various means: 1n numerous 
artworks; on a website devoted to the project; through GPS 
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onto smartphones for mobile users on site at the Bundanon 
property. 


The environmental portrait of Bundanon also incorporates 
the social and cultural history of the region, as it pertains to 
its environmental condition. Science meets art in the com- 
munication of environmental data through artworks and me- 
dia technology. Heavy Metal has been created as part of this 
process and invites us to interact with one of Arthur Boyd’s 
paintings to discover a hidden world of elements and miner- 
als in an experience that 1s simultaneously chemical, visual 
and musical. 


The initial concept for Heavy Metal arose after spending 
time on-site with Professor Mark Taylor. Mark and his stu- 
dents had been surveying the mineral composition of Bun- 
danon, looking for traces of human activity that, for exam- 
ple, derived from farm and workshop activities, but also 
looking for the effects of upstream mining for gold and other 
heavy metals. 


Elements and minerals lay buried in the landscape tracing 
diagrams of human activity. Specks of alluvial gold washed 
down to the floodplain from worked-out mountain mine 
shafts; existing as mineral auras that reveal the long-van- 
ished outlines of farm buildings and the telltale chemical 
fallout from workplaces. Arthur Boyd painted this (mineral- 
ised) landscape with colours that were themselves formu- 
lated from earthy compounds and exotic metals, milled to a 
fine paste in linseed oil and turpentine.* 


Our intention was to create a map of human activity based 
upon this forensic evidence. While we were taking environ- 
mental samples, Nigel invited Mark to visit Boyd’s painting 
studio and bring along his portable mineral analysis ma- 
chine? as it struck him that we may have a great opportunity 
for rethinking Boyd’s works. The starting point was that 
Boyd was situated in this landscape, painting the physical 
features, and using (or making himself) colours that were 
substantially minerals (originally extracted from the earth), 
thus forming a metaphorical circuit. Mark was surprised by 
the massive levels of heavy metals in the materials used by 
painters and was keen to collaborate — so we proceeded to 
analyse the mineral composition of the entire colour range 
that Boyd used and came up with a massive database of min- 
erals that corresponded to his palette. 





Figure 1. Chemical sampling of studio paints. Image Nigel Helyer. 
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Figure 2. Chemical sampling of studio paints. Image Nigel Helyer. 


The second stage was to sample the Steinway piano at the 
Bundanon homestead, note by note. Firstly, we recorded 
regular keystrokes; secondly, we recorded the reverberance 
of the sounding board resulting in one to two minute sound 
files per note. Working with another colleague, Jon Drum- 
mond, who is an expert in data sonification, we created a 
computer-driven audio-visual system able to read the video 
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stream from a camera facing Boyd’s unfinished painting Re- 
turn of the Prodigal Son (c1997). The screen interface dis- 
plays a highly magnified colour ‘target’ area from the paint- 
ing along with the RGB values and the predominant miner- 
als present, which are shown as elements of the periodic ta- 
ble. The system then translates the stream of mineral data 
into sound, which is layered in two components: a general- 
ised harmonic chord structure that corresponds to the colour, 
overlaid by individual note highlights that illustrate the dis- 
tribution of the most prominent minerals. The computer 
monitor gives feedback on the area of interest, colour ratios 
and a graphical display of the minerals detected. 





FIGURE 3. Jon Drummond sampling the Steinway Piano. Image 
Nigel Helyer. 


The Cultural Theorist Cecelia Cmielewsk1 invokes the pres- 
ence of Heavy Metal during its debut in the following lyrical 
extract from the catalogue to the Landscape/Portrait exhibi- 
tion that documented the When Science meets Art research 
project. 


Winding toward Boyd’s studio through the beautiful garden 
at Bundanon, with flowering azaleas and orange clivias, un- 
der the shade of bright-green deciduous amber trees, the 
chord-like sounds of a piano become increasingly distin- 
guishable. The sounds do not make a discernible melody 
but, once in the studio, they exert an immediate calming and 
almost meditative effect. Visitors are intrigued and de- 
lighted by this work. Occasionally a high note pops into the 
space that jolts the listener into attentiveness. The sound of 
a painting - Artists have responded to paintings by playing 
music to them, but not until Heavy Metal have they com- 
posed music from their material composition. 


This work is a genuine collaboration between scientist and 
artist. It is rare for such a collaboration to actually be a 
creative conjuncture of both disciplines. Usually one is at 
the behest of the other. Either the art is used to explain or 
‘communicate’ the science or the science is made too simple 
by the art. Helyer has a good grasp of many scientific 
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principles and has worked with scientists for over thirty 
years, the results of which we see in this collaboration. 


Heavy Metal is interactive at a complex and conceptual 
level. The composition of chord-like sounds (recorded from 
the homestead Steinway) is created by a real-time analysis 
of the minerals in the colours of an unfinished Boyd paint- 
ing, Return of the Prodigal Son (c1997). As a video camera 
is trained onto a section of the canvas, the screen displays 
the mineral content of the selected colours, in the form of 
the periodic table. The image and corresponding sound 
change each time someone Selects a new section of the can- 
vas on which to train the camera. Heavy Metal also brings 
together two kinds of science: environmental and computa- 
tional. 





FIGURE 4. In the studio with Elders from the Wreck Bay commu- 
nity. Image Nigel Helyer. 


The creative leap of the artist is matched with the precise 
methodology of the scientist. Heavy Metal could have been 
quite a cold work — simply a digital archaeology of a paint- 
ing. Instead, it is a lively work that uses the warm sounds of 
the piano and finely calibrated composition to bring the 
painting into a new space for contemplation. Heavy Metal 
provides participants with different ways to animate a 
‘static’ painting. It takes some time for viewers/listeners to 
put together what it is that they are experiencing. The sound 
is dynamic, based on the elements used in a particular area 
of Boyd’s oil painting. 


One little boy of about seven years old knows the periodic 
table. He is thrilled to be able to ‘read’ this painting because 
each sequence on the screen includes a representation of the 
elements from a particular section of the painting. All of a 
sudden he is able to correlate his knowledge of the periodic 
table with the materials used by the artist. For another visi- 
tor, a writer, who does not usually ‘get’ art, the layers of 
sound and data provide a way for her to consider the work 
beyond that of colour and texture. For some of the men vis- 
iting from the Wreck Bay community, the data resonates 
with their use of naturally occurring materials used to paint 
their bodies for ceremony. Everyone who comes to Site- 
works (and there is a large audience) spends time with 
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Heavy Metal. The studio stays open for several hours longer 
than scheduled, and many visitors come back more than 
once. As the sounds from the studio close down, the chorus 
of the frogs in the nearby lake take over in the dark of even- 
ing at Bundanon. 





FIGURE 5. The camera vision system, screen and target painting. 
Image Nigel Helyer. 


Conclusion - Art Science and the Environment 


Heavy Metal, draws upon recent research that has sought to 
develop collaboration between art and science, particularly 
in the context of technology and the environment. Sian Ede 
has proposed that there is ‘much in contemporary science 
that can stimulate art’s flexible, intuitive and visceral re- 
sponse to the world’. Ede also argues that ‘the fragile envi- 
ronment’ might well become ‘the most crucial matter for the 
future concerns of both artists and scientists’. Concern for 
the environment has become a central political and artistic 
issue in the contemporary world. 





FIGURE 6. Detail of the Screen showing target area and Periodic 
Table. Image Nigel Helyer. 


Recent publications have emphasised the heightened regard 
for the environment in ‘eco-aesthetics’, in ‘eco art in pursuit 
of a sustainable planet’, in ‘art and ecology now’ as manifest 
in ‘land art’ incorporating landscape, earthworks, environ- 
mental art, sculpture, and nature-based installation art. Sean 


Cubitt has argued that eco-politics 1s indeed “the single larg- 
est unifying political discourse of the early 21st century’ Cu- 
bitt suggests that artworks can voice the contradictions of 
their period, including the role of technology. It may be 
demonstrated that ‘not all technologies are instrumental, that 
is, used as instruments for domination over nature’. Media 
forms and art works may rely on certain technologies to 
communicate an ecological sensitivity. 


Oratorio for a Million Souls 


Introduction and context: Oratorio for a Million 
Souls - why Bees? 


Why Bees? — There are two main reasons for such interest 
— firstly Bees, along with other colonial insects (such as 
termites and ants) display striking social organisation — that 
since ancient times they have provided powerful metaphors 
for human social structures. Secondly Bees have become a 
focus species in the public debate about the environment. 
Unfortunately the focus upon the threat to Bees; on Colony 
collapse and the subsequent effect on food production fre- 
quently masks the even wider issues of diminishing Bio-di- 
versity in the face of industrial culture; agribusiness and cli- 
mate change. 





FIGURE 7. A Bombus Terrestris nest. Image Nigel Helyer. 


But to return to metaphor — historically the Bee has been 
and remains a symbol of good (hierarchical) government — 
with a vast population of obedient workers ruled over by a 
Queen. In English we still use the terms As busy as a Bee 
and A hive of industry as phrases that affirm diligence and 
application to a task. Until quite recently the reproductive 
cycle of the Bee was a mystery and the Queen was long re- 
garded by the Christian Church as analogous to the Virgin 
Mary; until 1t was discovered that she made a single but pro- 
longed nuptial flight mating with up to eighty males, retain- 
ing their sperm for the duration of her long productive life 
— not such a chaste Lady after all! 

The workers who make up the vast majority of the hive pop- 
ulation, have stood for loyalty, obedience, courage and 
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selflessness ~ and much of their behaviour would seem to 
support these anthropomorphic metaphors we attach to them 
— perfect role models for Nineteenth Century industrial 
capital, which like the Monarchy is based upon an autocratic 
power pyramid. 


However, recent research turns this pyramid image upon its 
head. Consider for a moment the behaviour of bees in 
Swarming mode. The hive has grown and the colony di- 
vides. The outgoing group muster, hanging from a branch 
somewhere, seeking a new location. The swarm sends out a 
stream of scouts, often over a period of days, who report 
back, using methods similar to the bee dance to relay com- 
plex qualitative information. Somehow a collective process 
is engaged, the swarm considers this growing matrix of spa- 
tial data and eventually they fly to the most favoured loca- 
tion to begin a new colony. This is not the work of an indi- 
vidual mind, it is a product of parallel processing, a natural 
neural network, if you like, that has evolved over a 100 mil- 
lion years, the hive as super-organism. 
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FIGURE 8. The Oratorio at Buitenpost Gardens. Image Nigel 
Helyer. 


European Capital of Culture Leeuwarden 2018 


The Oratorio for a Million Souls® was commissioned by the 
European Capital of Culture, Leeuwarden 2018 (Nether- 
lands). Funded by a European Union Inter-Regional pro- 
gramme the project has been realised 1n three botanical gar- 
dens; Dr Kruidhof in Buitenpost (Netherlands) and in Em- 
den and Oldenburg (Germany). Each site consists of a 
‘green-architectural’ listening space that houses two intelli- 
gent bee-hives to produce a four channel immersive sound- 
scape in realtime. 


The Oratorio for a Million Souls was developed in conjunc- 
tion with a citizen science project in which one hundred sen- 
sor equipped beehives were distributed to schools in the 
Netherlands and Germany with the aim of mapping and as- 
sessing the pollen load (or floral productivity) of local ecol- 
ogies. This research is taking place in a context which 1s 
commonly referred to as the European “Green Desert” a 
landscape which at first glance appears to be verdant and 
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productive, but which in reality is the product of high-inten- 
sity industrial agriculture that is effectively eliminating bio- 
diversity and causing significant reductions in native species 
of birds and insects. 


The concept behind Oratorio 1s to create a network of bio- 
acoustic listening spaces that allow visitors to immerse 
themselves in a world of insect sounds. By entering into the 
sonic heart of a Bee-city a visitor can for a moment become 
a fellow citizen — meditating at the centre of a sonic me- 
tropolis; all working; all buzzing — and taking an empa- 
thetic leap that we hope permits some serious thoughts about 
our collective effects on the natural environment. 


Physically each Oratorio is constructed in traditional wattle 
and daub, intricately hand woven willow branches plastered 
with a thick layer of Leem’ with a form based upon the shape 
of a traditional woven straw beehive (a Skepp in English). 
The structures have a thatched roof reminiscent of the tradi- 
tional farming buildings of the region, and this in turn sup- 
ports a green-roof — sown with Bee attracting flowers — a 
sort of Bee-Helipad! The hemispherical interior of the Ora- 
torios provide a special acoustic, similar to that found in the 
domed roofs of Cathedrals and Mosques where sound looses 
its directionality and forms an omni-directional soundcloud. 





FIGURE 09. The Oratorio concert. Image de Kruidhof Buitenpost. 


Each of these green architectures contains two sensor hives 
furnished with colonies of Bombus Terrestris the common 
Bumble Bee®. We employ high definition miniature DPA 
microphones’ in the Bee entry ports that are capable of re- 
cording the sounds of individuals as they arrive or depart the 
nest; as well as a set of large scale piezo contact micro- 
phones in the nest area that record the collective activity of 
the hive. These two streams of live audio are mixed with 
sonifications of Bee entry and exit data which are rendered 
as short arpeggios of piano notes; rising upon exit and de- 
scending upon arrival, and which become more complex 
with the frequency of Bee activity. The arpeggios provide 
visitors with a direct sonic index of the intensity of Bee 
flight activity as this can only be seen from the exterior or 
the Oratorio. Hive temperature 1s rendered as a series of sus- 
tained tones that rise or fall as the temperature in the hive 
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varies. The outputs of the two hives are combined into a four 
channel live soundscape which is augmented by a screen 
displaying realtime data of bee activity and data sonifica- 
tion. 


The province of Fryslan in the Netherlands and the adjacent 
area of OstFriesland in Germany boast many talented town 
Brass Ensembles and we were also commissioned to create 
a series of musical scores derived from the sounds and data 
sets recorded in the Oratorios, that could link each of the 
sites. By analysing the harmonic range of hive recordings 
we derived a tonal palette that we quantised into an equal 
tempered scale. Likewise by analysing the Bee entry and 
exit data from a range of the citizen science hives we estab- 
lished a temporal or rhythmic framework as well as devel- 
oping an overall compositional structure based upon the di- 
urnal activity of a hive (which is related to light and temper- 
ature). 





FIGURE 10. B_Rhapsodie in performance. Image Nigel Helyer. 


We created a musical event in which three the Brass ensem- 
bles located in Buitenpost, Emden and Oldenburg, collabo- 
rated to play a multi-part score via a satellite Video and Au- 
dio link-up — a task not without technical or aesthetic chal- 
lenges. 


Creative Challenges - The Oratorios 


Working with living biological systems is always complex 
and unpredictable - Bees especially so in terms of handling 
nests of flying (and potentially stinging) insects who do not 
like being disturbed! This made the work of testing and 
mounting microphones inside the hives a physical chal- 
lenge. Acoustically the project also had to develop solutions 
to prevent feedback within the sound architecture, which 
contained the two hives as live acoustic sources. The highly 
sensitive DPA microphones embedded in the Bee entry/exit 
ports provide an extremely detailed audio stream of flight 
and crawling sounds and as they are located on the building 
exterior are not affected by the interior soundscape. In con- 
trast our early experiments with conventional microphones 
placed inside the hive boxes were a complete failure due to 
persistent feedback. The interior of the nests produce a range 
of relatively quiet, crawling, peeping and contact events 
which when employing conventional studio microphones 
require too much amplification; the hive boxes themselves 


resonating with the sound energy from the four channel 
speaker array. The solution was provided by wiring together 
two large format Piezo discs and mounting them onto a 
small sounding board placed onto the upper surface of the 
nest enclosure. This extremely low-tech approximation of a 
contact microphone delivered good sound fidelity without 
the problems of feedback. 
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FIGURE 11. A BumbleBee about to enter the Orato 
gel Helyer. 
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Creative Challenges - The Musical Scores 


From the perspective of traditional musical aesthetics both 
the temporal and pitch structures of the Bee audio and data 
derived composition are unconventional. Whilst the three 
Brass Ensembles are musically extremely competent but as 
our scores do not resemble their normal repertoire this gen- 
erated some initial resistance — the score is certainly not an 
imagistic representation of the life of a hive (a la Flight of 
the BumbleBee) but rather an indexical translation of that 
life and activity. 


The debut of the composition B-Rhapsodie took place 
during June 2018 across the three sites. The collaborative 
rendition of the score by three geographically distant Brass 
Ensembles was a logistically difficult operation. We had to 
carefully synchronise the musicians and deal with the inev- 
itable time lag in satellite transmission. Each site could see 
and hear the other two distant ensembles — and somehow, 
despite cloudbreaks of cold European summer rain trickling 
into Tubas and English Horns the concert was a success! 


Legacy 

In line with the objectives of the European Capital of Cul- 
ture, works included in the programme are designed to have 
a legacy that benefits and extends the cultural life of the re- 
gion. The Oratorio for a Million Souls project and its linked 
citizen science collaboration have an ongoing life in terms 
of research; public advocacy (for the preservation and 
growth of Bio-diversity) and in the very palpable form of 
the three Oratorios in their respective botanical gardens 
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which are designed to function for at least three years. Each 
botanical garden is free to incorporate the Oratorio into its 
visitor and education programme giving a new and extended 
meaning to the authors original intentions. 


Endnotes 


[1] Heavy Metal archive 
http://www.sonicobjects.com/index.php/pro- 
jects/more/heavy_ metal 

[2] The Bundanon Trust NSW Australia https://bundanon.com 

[3] Arthur Boyd; One of the first Australian painters to be widely 
recognised internationally, specifically in the United Kingdom as 
one of the Antipodeans. 

[4] Helyer quote from the Landscape/Portrait catalogue 2017. Pub 
Macquarie University. 

[5] To generate the data programmed into Heavy Metal, we used a 
hand-held X-ray fluorescence spectrometer. 

[6] Oratorio for a Million Souls archive 
http://www.sonicobjects.com/index.php/projects/more/orato- 
rio_for_a million souls 

[7] Leem - a mixture of clay, sand and chopped straw. 

[8] The bee nests we used were supplied commercially. In Holland 
much of the agriculture takes place in vast green-houses and the 
pollination is carried out by Bumble bees who are ‘buzz’ pollina- 
tors — using their wing muscles to vibrate pollen from the flower. 
[9] DPA 4060 omnidirectional lavalier microphones 
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